feat(ingest): support setting fields for csv without header row + csv batching #221
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lukasmalkmus
force-pushed
the
lukasmalkmus/csv-fields
branch
from
553727b
to
556a37a
Compare
lukasmalkmus
force-pushed
the
lukasmalkmus/csv-fields
branch
2 times, most recently
from
91ddd77
to
0cfc4c2
Compare
auto-merge was automatically disabled
April 26, 2024 10:29
Pull request was converted to draft
lukasmalkmus
force-pushed
the
lukasmalkmus/csv-fields
branch
from
0cfc4c2
to
fef3cf0
Compare
lukasmalkmus
force-pushed
the
lukasmalkmus/csv-fields
branch
2 times, most recently
from
49d7255
to
f99e760
Compare
bahlo
reviewed
May 2, 2024
lukasmalkmus
force-pushed
the
lukasmalkmus/csv-fields
branch
from
f99e760
to
82a2a00
Compare
lukasmalkmus
force-pushed
the
lukasmalkmus/csv-fields
branch
from
82a2a00
to
9b3f76e
Compare
bahlo
approved these changes
Aug 20, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
CSV should be batchable as well!
Also adds the CSV headers feature for CSV files without a header row.
In line with axiomhq/axiom-go#305, I also addressed the batching behaviour here and made it less aggressive in time (flush batch every 5s instead of every 1s) but also introduced batch size based flushing (every 10k), whatever condition is met first. Also files are now batched instead of only
stdindata.